SAU: Smooth Activation Function Using Convolution with Approximate Identities

AbstractWell-known activation functions like ReLU or Leaky are non-differentiable at the origin. Over years, many smooth approximations of have been proposed using various smoothing techniques. We propose new a function by convolving it with approximate identities. In particular, we present and show that they outperform several well-known in datasets models. call this Smooth Activation Unit (SAU). Replacing SAU, get 5.63%, 2.95%, 2.50% improvement ShuffleNet V2 (2.0x), PreActResNet 50 ResNet models respectively on CIFAR100 dataset 2.31% (1.0x) model ImageNet-1k dataset.KeywordsSmooth FunctionNeural Network